Using Large Page and Processor Binding to Optimize the Performance of OpenMP Scientific Applications on an IBM POWER5+ System

Multicores are widely used for high performance computing and are being configured in a hierarchical manner to compose a multicore system. While this presents significant new opportunities, such as high inter-core bandwidth and low inter-core latency, it also presents new challenges in the form of inter-core resource conflict and contention. A challenge to be addressed is how well current shared-memory parallel programming paradigms, such as OpenMP, exploit the potential offered by such a multicore system for scientific applications. In this paper, we analyze the performance of OpenMP scientific applications such as NAS parallel benchmarks, an OpenMP Matrix multiplication, and a large-scale scientific application: a 3D particle-in-cell application Gyrokinetic Toroidal Code (GTC) in magnetic fusion on an IBM POWER5+ system, and use large page and processor binding to significantly optimize the OpenMP performance for no requirement of any program modifications. Our experimental results show that using the large page of 64KB on the multicore system results in up to 33.92% performance improvement for OpenMP NAS parallel benchmarks, and the OpenMP benchmarks benefited more from the large page than their MPI counterparts; using the large page of 64KB and processor binding results in up to 67.18% performance improvement for matrix multiplications, and up to 10.13% performance improvement for the GTC. Our results also indicate that the OpenMP performance can be improved by using conventional loop optimization techniques such as blocking and unrolling inside the OpenMP parallel regions.